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Abstract — Current methods for evaluating the effects of human 
opinions in data fusion systems are often dependent on human 
testing (which is logistically hard and difficult to arrange for 
repeated tests of the same population). The alternative is to use 
hypothetical examples, which tend to be simplistic. To facilitate 
studies of data fusion architectures which integrate “soft” human- 
generated decisions, we have used a simulator of subjective 
beliefs. The simulator is based on the two-stage dynamic signal 
detection model of Pleskac and Busemeyer (2010). We use this 
scheme to simulate human opinions and combine them using 
belief fusion methods, including Bayes’ Rule; Dempster’s Rule 
of Combination (DRC); Yager’s rule; the Proportional Conflict 
Redistribution Rule #5 (PCR5) from Dezert-Smarandache the- 
ory; and the consensus operator from subjective logic. In our 
simulations, the DRC and Bayes rule exhibited performance that 
was on par with, and in some cases better than PCR5 and the 
consensus operator (when used in conjunction with a measure of 
source reliability). In all simulated cases, Yager’s rule exhibited 
inferior performance. 

I. Introduction 

The integration of subjective data sources in a data fu- 
sion system (also known as “soft fusion”) has been studied 
intensely over the past few years [1], In addition to the 
“hard fusion” of sensory information from devices that use 
electronic, optical, and acoustic modalities, there is a grow- 
ing interest in augmenting decision making with available 
“soft” sensors (i.e., human opinions and assessments [1]). 
The incorporation of human opinions into data fusion systems 
could potentially improve accuracy and reliability. However, 
human opinions are difficult to model as they do not exhibit 
fixed error probabilities, and are often not easily characterized 
by probability distributions [1], Furthermore, the employment 
of large numbers of humans for testing can be logistically 
challenging and expensive, and opportunities to re-test the 
same humans on modified data presentations and exposition 
schemes is often difficult. At least in the early stages of testing 
and tuning of data fusion algorithms, it may be desirable to 
use models of human decision making rather than using actual 
human-generated data. 

Much work has been devoted over the past fifty years to 
developing new ways of representing and combining subjec- 
tive and imprecise beliefs in areas such as pattern recognition, 
biometrics, medical diagnostics, and autonomous navigation. 
However, the majority of studies which include elements of 
human decision making have resorted to hypothetical ex- 
amples, observing how fusion methodologies perform with 



respect to how a ’’logical and coherent human” would reason 
[2] — [4]. Other studies have adopted simple stochastic models, 
observing the performance of combined decisions through 
Monte Carlo simulations (e.g., confusion matrices [5]). There 
have also been several attempts to amass groups of humans 
for direct testing and analysis [6]. This last approach is in 
many ways preferred to the models it has replaced, but is 
logistically cumbersome and somewhat inflexible, especially 
in assessing systems and algorithms that require the tuning of 
a large number of parameters. 

Models of human decision making from the social sciences 
and cognitive psychology have not been applied extensively to 
soft and hard/soft fusion systems. The few studies that have 
used such models tend to look at how task reward structures in- 
fluence human decision-making strategies in situations where 
the human element acts in a supervisory role [7]. The present 
study seeks to analyze soft fusion systems where subjective 
data sources provide confidence assessments of the decisions 
they make. In Section II, we overview the model for human 
simulation used here, known as two-stage dynamic signal 
detection (2DSD). In 2DSD, humans are modeled via a tuple 
of parameters which direct a stochastic process that represents 
an internal evidence accumulation between two outcomes. The 
human tuples used here are taken from [8], and are the result 
of modeling human responses in a line length discrimination 
task , where human subjects are shown different pairs of lines 
and asked to identify (1) the longer of the two lines and (2) rate 
their confidence in their response on a subjective probability 
scale. In Section III, we review a few methods for representing 
and combining subjective beliefs. In Section IV, we describe 
ways of mathematically formulating human opinions for use 
with the belief combination methods of Section III. Finally, 
in Section V we use a subset of the data from [8] to simulate 
human opinions for the line length discrimination task and 
fuse them using the methods described in Sections III and IV. 

II. Human Simulation Methodology 
A. Two-Alternative Forced Choice Tasks 

Human decision-making models have been a topic of inter- 
est for psychologists since the early 1960s [9]. The majority of 
work has been addressing decision making in two-alternative 
forced choice (TAFC) tasks, in which a subject is presented 
with a scenario and is forced to choose between two alterna- 
tives [9]. Models of decision making based on TAFC tasks 
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assume that ( 1 ) internal evidence favoring each alternative is 
accumulated over time; ( 2 ) the process of internal evidence 
accumulation is subject to random fluctuations; and (3) a 
decision is made when a sufficient amount of internal evidence 
has been accumulated for one of the two alternatives [9]. As 
described in [ 8 ], many models of human decision making have 
been based on Dynamic Signal Detection (DSD). 

Definition 1: Dynamic Signal Detection (DSD) [8], Consider 
a TAFC task given by the choice A and its negation A. Let the 
real-valued function L(t) represent the accumulated evidence 
in favor of A 1 up until time instant t. A Dynamic Signal 
Detection (DSD) model is described by the stochastic linear 
difference equation 

AL(t) = SAt + V~Kt eft + At), L(0) = L 0 (II. 1 ) 

where 5 & ft is known as the drift rate and e(t) is a white 
noise process with zero mean and variance a 2 . The value a is 
known as the drift coefficient. The drift rate 6 is either positive 
or negative, depending on whether A or A is true. To make a 
choice, the process occurs until a threshold, either 9 a, 9^ £ ^ 
is crossed (where —6 ~a < Lq < 9a)- The decision rule A[L(f)] 
is given as 

( A L(t) > 9a, 

A [L(t)} = l A L(t) < -0* (IL2) 

[wait otherwise. 

By itself, DSD does not present a method for modeling 
subjective confidences. In two-stage dynamic signal detection 
(2DSD) [ 8 ], a DSD process is simulated until a decision is 
reached after time td • The evidence accumulation process of 
Equation II.l is then continued for an additional r seconds, 
after which the value of L(fd + r) is binned into a set of 
confidence intervals depending on what state was chosen at 
time td- 

Definition 2: Two-Stage Dynamic Signal Detection (2DSD) 
[81. Consider a DSD model describing an internal evidence 
accumulation L(t) as defined for a TAFC as in Definition 
1. Suppose a decision of to £ { A , A} is made at time td- 
Let = \j)-f ] ■ ■ -pffl] denote the N u possible confidence 
values associated with choosing u>. The assigned confidence 
level p £ associated with deciding ui after waiting t c = 
td + r is given as 

p = p { f' } when L(t c ) £ (II.3) 

where 1 = — oo and c ^ = oo for each lo £ {A, A}. The 
remaining confidence bin parameters C = [c^ ■ ■ ■ c^^_ 1 ] 
are chosen such that Cj_i < c,; for each i £ {1, . . . , N u — 1}. 

Using Definitions 1 and 2, a human decision maker is repre- 
sented via the 2(Na + N-ff) + 4 parameters as summarized in 
the tuple S, 

S = { S , a, L 0 , 9 a, Oa, t, C (4 > , C® , P< 4 ) , P^}. (n.4) 

For a full description of these parameters, see [ 8 , Table 2], 

*L(t) < 0 represents evidence in favor of A. 



B. 2DSD Simulator Implementation 

The present study used data from [ 8 ] to simulate the fusion 
of opinions from six ( 6 ) subjects under a line length discrimi- 
nation task. In the line length discrimination task described in 
[ 8 ], subjects were shown one of six possible pairs of horizontal 
lines. For any given pair, the subjects were asked to identify 

( 1 ) which of the two lines was longer; and ( 2 ) rate their 
confidence in their response on the subjective probability scale 
{50%, 60%, . . . , 100%}. Six different line length pair types 
were presented, representing six different levels of difficulty. 
The work in [ 8 ] fit the subjects of the line length discrimination 
task using the 2DSD model parameters of Equation II. 4 with 
the following additional restrictions: ( 1 ) the decision thresh- 
olds for both choices were set to be equal (i.e., 9-^= 9a = 9); 

(2) the drift rate 5 was chosen on a per trial (or simulation) 
basis from N(u,r] 2 ); (3) the initial condition L 0 was chosen 
on a per trial (or simulation) basis uniformly in the interval 
[— 0.5 s 2 , 0.5s z ], where s z £ 5i + was the size of the interval; 
(4) the confidence bins for both choices were set equal (i.e., 
C 1 4 ' = C ( 4 1 = C); (5) the confidence values were fixed for 
each subject as P 1 "' 4 -* = p( 4 ) = [0.50, 0.60, •• • ,1.00]; and 
( 6 ) the drift coefficient was fixed for each subject as o = 0 . 1 . 
Expression II. 4 was thus reduced to the 10-tuple, 

S = {u, 7], S z , 9, T , Cl, C 2 , c 3 , c 4 , C 5 }. (II.5) 

The time step of the simulator was fixed at At = 0.001 
for each subject. The parameter v was allowed to vary with 
task difficulty. Specifically, the values and in, of [ 8 , Table 
6 ] represent the mean drift rates for a subject under the 
hardest and easiest task difficulties, respectively. The human 
simulator algorithm for a given S is shown in Figure 1. The 
parameter values used to simulate each subject can be found 
in [ 8 , Tables 3 and 6 ], Also, in [ 8 , Table 6 ] separate decision 
thresholds 9 were determined for two cases of the line length 
discrimination task: ( 1 ) when subjects were asked to focus on 
fast responses; and ( 2 ) when subjects were asked to focus on 
accurate responses. In the present study, the values of 9 which 
represent the subjects focusing on accurate responses were 
used. For brevity’s sake, the forthcoming simulation example 
(Section V) simulates opinions for each subject using the 
hardest task difficulty only (oi). 

III. Belief Fusion Methodologies 
A. Bayesian Epistemology 

Bayesian epistemology is the process of using the Kol- 
mogorov axioms to represent logical constraints on degrees 
of belief, and probabilistic conditioning as the primary means 
of inferential reasoning [10], [11]. Let fi represent a set of 
logical sentences which fully characterizes some phenomenon 
and IFa represent a set algebra defined over f l. The probability 
function P(ut) represents the degree of belief a source holds 
towards the logical sentence uj £ Cl. For inferential reasoning 
between multiple belief assessments, a popular fusion method 
is Bayes’ rule. 

Definition 3: Bayes’ Rule. Suppose we observe a possible 
piece of evidence x £ X related to the logical sentences 
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Require: S is a human source 10-tuple (expression II. 5) 


1 


function 2DSDHumanSim(<S) 


2 


Choose d from N(o,ij 2 ) 


3 


Choose Lq from U(— 0.5s 2 , 0.5s z ) 


4 


At <- 0.001, 


5 


<7 0.1 


6 


t<- 0 


7 


L ■£- Lq 


8 


while — 6 < L < 9 do > Simulate a DSD process 


9 


Choose e from N(0,a 2 ) 


10 


A L £- 6 At + e\f/At 


11 


L <r- L + AL 


12 


t £- t + At 


13 


end while 


14 


if L > 6 then 


15 


oj ■£- A > A is the decided state 


16 


else if L < —6 then 


17 


uj •*— A o A is the decided state 


18 


end if 


19 


td <— t 


20 


t c 4— I'd + 7" 


21 


while t < t c do > Simulate interjudgment time 


22 


Choose e from N(0,a 2 ) 


23 


AL 4 — 6 At + e\//At, 


24 


L <r- L + AL 


25 


t <- t + At 


26 


end while 


27 


Co i OO 


28 


Cq 4 — OO 


29 


P = [0.50,0.60,0.70,0.80,0.90,1.00] 


30 


for i - 1 — > 6 do > Confidence value binning 


31 


if Cj_i < L < Cj then 


32 


p -s— pi > Extract the ith element from P 


33 


end if 


34 


end for 


35 


return (uj,p) 


36 


end function 



Fig. 1 . Algorithm for implementing the 2DSD human simulator of [8] from 
the parameters given in Equation II. 5. 



described by all u> £ ft Our original (a priori) beliefs, Pq(lu), 
can be updated via Bayes’ Rule: 



Pn(w\x) 



P x (x\uj)Pq(uj) 

T,uen p x( x \u) p n(u) 



(III.l) 



where P x (x\u>) is known as the likelihood that the sentence 
u> £ fl would beget the evidence x £ X. The quantity Pq(w \x) 
is known as our a posteriori belief in the logical sentence u>, 
given our observation of the evidence x. 



consists of disjoint elements. Hence, the algebra on f l becomes 
the powerset Pci = 2 n . In DS Theory, one is able to define 
one’s beliefs across the powerset of f l through a belief mass 
assignment. 

Definition 4: Belief Mass Assignment (BMA). A function 
in : — > [0, 1] is a belief mass assignment for some frame 

fl if and only tf^2 X (zn m ( X ) = 1. 

As in [1 1], we assume that m(0) = 0. A BMA can be thought 
of as a normalized measure of the explicit evidence a source 
holds towards each subset of ft As opposed to probability 
functions, BMAs can be used to specify evidence on a set 
of sentences A C 52 without imposing restrictions 2 on the 
individual sentences ut £ A. 

For a given set of sentences A C Cl, it makes sense to define 
the total amount of evidence assigned to A and the amount 
of evidence which does not contradict A. These quantities are 
known as belief and plausibility respectively. 

Definition 5: Belief/Plausibility Functions. Consider the 
BMA m on some frame f l. The functions Bel : 2 s1 — > [0, 1] and 
PI : 2 s2 -A [0, 1] are known as belief and plausibility functions 
on ft and are given for any A C fl as 

Bel (A) = Y m ( x ) (III.2) 

XCi 

and 

P1(A) = 1 - Bel(A) = Y "iPO- (III. 3) 

xn AyUi 

The difference PI (A) — Bel(A) describes the amount of un- 
certain evidence (or imprecise evidence) held towards A. 

Definition 6: Uncertainty Function. Consider the BMA m 
on some frame ft The uncertainty function Un : 2^ — > [0, 1] 
is given as 

Un(A) = P1(A) - Bel(A) = Y m ( x )- (III. 4) 

xn A^H 
xg :a 

A few special types of belief functions are relevant to the 
present study. A vacuous belief function is one which repre- 
sents total ignorance towards f l such that 

fl A = ft 

ml A) = ’ (III.5) 

v ; |0 A^n. 

A simple support function assigns a degree of belief s £ [0, 1] 
to some A* C fl such that 

(s A = A*, 

m(A) = l 1 - s A = fl (III.6) 

I 0 otherwise. 



B. Dempster-Shafer Theory 

The limitations of Bayesian epistemology are summarized 
in [10] and [11, Chapter 1], One attempt to develop an 
alternative is given in Dempster-Shafer (DS) Theory [11], 
Consider a set of logical sentences fl and a corresponding 
set algebra Pq. For simplicity, we assume that fl is finite and 



It has been shown that Bayesian probabilities form a subset 
of belief functions. More specifically, a belief function whose 
corresponding BMA is non-zero for only the singleton ele- 
ments of 2 n is a Bayesian probability function [11, Theorem 
2.8], It is also easy to see from the definitions of belief 

2 For examples of these restrictions, see [11, Chapters 1-2] 
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and plausibility that Bel(4) < PI (4) for any 4 C 11, with 
equality only when Un(4) = 0. This relationship leads to 
a set of intervals [Bel(4), Pl(4)] for every 4 C 11. These 
intervals beget a class of probability assignments, which can 
be thought of as possible subjective probabilities which the 
evidence supports 3 . 

Shafer cites Dempster’s Rule of Combination (DRC) in [11] 
as the primary method for belief aggregation and updating. 

Definition 7: Dempster’s Rule of Combination (DRC) [11]. 

Consider the opinions of two sources, represented by the BMAs 
mi and m 2 respectively. Dempster’s Rule of Combination is 
defined as an orthogonal combination of these belief functions. 
The resulting BMA mi j2 (4) for any A C Cl is given by 



C. Variations on Dempster-Shafer Theory 

If the combined sources present highly conflicting opinions 
and it is not feasible to perform source discounting, a DS 
theoretic approach may not provide an acceptable result [2], 
[14], One variation. Yager’s Rule, calls for interpreting the 
evidence in conflict as uncertain evidence. 

Definition 10: Yager’s Rule [15]. Consider two BMAs m \ and 
m 2 over some set of logical sentences O. The resulting BMA 
m 1,2 after combination via Yager’s rule for any A C Cl is 
given as 

{ Eai,a 2 cq m 1 (A 1 )m 2 (A 2 ) Af=Cl 

A 1 nA 2 =A (III. 11) 

mi(4)?n 2 (4) + K, A = fl 



rat, 2 (4) = f * ) Y m 1 (A 1 )m 2 (A 2 ) (III.7) 

' ' Ax.AaCfi 

AinA 2 =A 

where K < 1 is the measure of conflict after combination, 
given by 

JC = Y TO i(4 1 )m 2 (4 2 ). (III. 8) 

Ai ,A 2 CQ 
A 1 nA 2 =di 



Since the DRC is both commutative and associative, extending 
it to the combination of more than two BMAs is straightfor- 
ward. 

Up to this point, DS Theory has not addressed how to take 
into account both source confidence and source reliability at 
the same time. Shafer suggests in [11, Chapter 11] a method 
for altering a BMA based on a level of perceived reliability 
known as source discounting. This notion was later redefined 
in [13] in terms of a BMA as follows. 

Definition 8: Discounting on BMAs [13], Let a £ [0, 1] 

represent the reliability of a source. Source discounting the 
BMA m for every A CD is given by 



m(A ; a) 



am(A) A f=- 12, 

am(A) + (1 — a) 4 = S 1. 



(HI-9) 



Source discounting can also be performed for Bayesian prob- 
abilities as follows. 



Definition 9: Discounting on Bayesian Probabilities. Let a £ 

[0, 1] represent the reliability of a source. Source discounting 
the Bayesian probabilities P for every ui £ Cl is given by 

P(w; a) = aP{u>) + |12| _1 (1 — a) (III.10) 



where |12| represents the cardinality of Cl. 

Source discounting for Bayesian probabilities can be thought 
of as increasing the level of entropy in the source’s belief 
proportional to a. For a fully discounted Bayesian probability, 
all outcomes are made equiprobable (the analogue of the 
vacuous BMA in a DS theoretic construction). 



Probabilities defined in this way are referred to as “probabilities of 
provability” and in general do not carry a statistical interpretation [12] 



where K, is the degree of conflict between m\ and m 2 as given 
in expression III. 8. 

Yager’s rule is commutative but in general not associative. 
It is also possible to subdivide evidence amongst conflicting 
propositions proportionate to the degrees of belief held by 
the sources before combination. This concept has lead to the 
family of Proportional Conflict Redistribution (PCR) rules by 
Dezert and Smarandache. In this paper we investigate PCR5, 
the PCR rule most widely used at present [16], 

Definition 11: Proportional Conflict Redistribution Rule #5 
(PCR5) [16]. Consider two BMAs m.\ and m 2 over some 
set of logical sentences Cl. The resulting BMA mi, 2 after 
combination via PCR5 for any A C 0 is given as 



rai, 2 (A) = Y rai(Ai)m 2 (A 2 )+ 

Ay ,A 2 CQ 
A \ C\A 2 —A 

y- f m 1 (X) 2 m 2 {A) 

Vrai ( x )+m 2 {A) 

A'n 4=0 

TO 2 (4T) 2 mi(A) \ 

m 2 (X) + mi (4) / ’ 



(III. 12) 



Similar to Yager’s Rule, the PCR5 is commutative but in 
general not associative [16]. Finally, subjective logic [17] 
uses the general terminology of BMAs, belief functions, and 
uncertainty while providing a way of representing any Cl as a 
binary frame focused on some 4 C Cl and its negation. The 
result is a four-valued vector known as an opinion tuple, which 
can be rephrased in terms of BMAs for a binary 12 as follows. 



Definition 12: Consensus Operator, Binary Frames [17]. 

Consider the binary frame Cl = {4,4}. The consensus 
operator mi 2 can be represented completely in terms of the 
BMAs m\ and m 2 as 



rai, 2 (4) 

rai, 2 (4) 



rai, 2 (4 U 4) 



mi(4)m 2 (4 U 4) + m 2 (4)mi(4 U 4) 
ICc 

mi(4)m 2 (4 U 4) + m 2 (4)mi(4 U 4) 



ICc 

mi(4u 4)?n 2 (4 U 4) 
ICc 



(III. 13) 
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where ICc = mi(A\J A) +m 2 (AU A) — mi(AU A)m 2 (A\J A). 
If ICc = 0 , then the consensus operator becomes 

mi(A) + m 2 (A) 



mi j 2 (A) = 

^ m 1 {A) + m 2 (A ) 

mi, 2(A) = , 



(III. 14) 



mi >2 (A U A) = 0. 

The consensus operator is commutative and associative [17]. 



IV. BMA Formulation Methods 

According to the algorithm of Figure 1, a simulated human 
opinion produces a decision oj £ {A, A} and a confidence 
value p £ [0,1]. To use the combination operators described 
in Section III, oj and p must be translated into BMAs (or in 
the case of Bayes’ Rule, probability functions). In the present 
study, oj and p were used to formulate simple support functions 
of degree s focused on oj, as in expression III. 6 . For Bayes’ 
rule, the likelihood values of Equation III. 1 were formed such 
that Px[x\oj) = s and Px{x\uj) = 1— s. The degree of support 
s was set in the following four ways 

1) Decisions Only: Each subject’s source strength s was 
estimated beforehand by determining the subject’s pro- 
portion of correct decisions over 2,000 simulations. The 
subject’s simulated confidence value p was unused. 

2) Confidences Only: Each subject’s source strength s was 
taken as the simulated confidence value p produced at 
each run of the human simulator. 

3) Decision Reliability Discounting: Each subject’s source 
strength s was taken as the simulated confidence value 
p produced at each run of the human simulator. The 
resulting simple support function was then discounted 
using expression III. 9 (or in the Bayesian case, ex- 
pression III. 10). The discount rate a was taken as the 
subject’s proportion of correct decisions as described in 
the “Decisions Only” BMA formulation method. 

4) Evidence Strength Discounting: Each subject’s source 
strength s was taken as the simulated confidence value p 
produced at each run of the human simulator. The result- 
ing simple support function was then discounted using 
expression III. 9 (or in the Bayesian case, expression 
III. 10). The discount rate a was taken as the subject’s 
average evidence strength, estimated beforehand over 
2,000 simulations. The evidence strength £(oj, p) of a 
subject’s decision oj and confidence assessment p was 
given as 

£(oj, P ) = 1 - BS{oj, P \oj*), (IV.l) 

where oj* is the true outcome and BS(oj,p\oj*) is the 
quadratic scoring rule known as the Brier score [ 8 ]. 



Definition 13: Brier Score [8]. Let p £ [0, 1] be a con- 
fidence assessment towards the outcome oj £ {A, A}. Let 
oj* £ { A , A} represent the true outcome. The Brier score is 
a measure of distance on the interval [ 0 , 1 ] between the given 
confidence assessment and a confidence assessment which 
assigns the true outcome probability one, given by 



BS(oo,p\oj*) 



(1 -P? 
2 



P 



OJ = OJ*, 
OJ 7 ^ OJ * . 



(IV.2) 




Fig. 2. Evidence strengths of fused subjective data on the line length task, 
hardest difficulty. Clear bars indicate the evidence strength ranges for each 
fusion method. Higher evidence strengths and smaller evidence strength ranges 
are better. 



V. Simulations 

The BMA formulation methods given in Section IV and 
the combination operators described in Section III were used 
to perform fusion on the simulated subjects of [ 8 ]. All belief 
combination methods were initialized using vacuous BMAs 
as defined in expression III.5 (or in the case of Bayes’ 
rule, equiprobable outcomes). All BMA formulation methods 
were simulated for each combination method over 2,000 mns. 
Performance for Bayesian case was taken as the average com- 
bination evidence strength £(u)*,ps a j /es |w*), where PBayes 
was the resulting a posteriori probability after combination 
via Bayes’ rule. For the belief function combination methods, 
ranges on combination evidence strength were generated using 
belief and plausibility values as lower and upper bounds on 
possible subjective probabilities. The size of the evidence 
strength range represents the precision of the combination rule, 
and the lower bound of the evidence strength range represents 
the accuracy of the combination rule. An ideal combination 
rule should have high accuracy (i.e., evidence strength close to 
one) and high precision (i.e., a small evidence strength range). 

Figure 2 shows the average evidence strength of each com- 
bination method (namely, Bayes’ Rule, DRC, Yager’s Rule, 
PCR5 and the Consensus operator), across the four BMA for- 
mulation methods discussed in Section IV (namely. Decisions 
Only, Confidences Only, Decision Reliability Discounting, and 
Evidence Strength Discounting). For comparison, the highest 
and lowest average evidence strengths among the six subjects 
are also shown. Subjects were combined in groups of two in 
the order (((<Si © S 2 ) © S 3 ) ©•••)© S$), where the symbol © 
represents any of the combination rules from Section III, and 
the subscript of S represents the subject number as given in the 
columns of ( 8 , Table 6 ]. Both Bayes’ rule and the DRC were 
unable to combine subjective confidences consistently without 
performing source discounting. In all other cases, Bayes’ 
rule and the DRC produced combination evidence strengths 
on par with the PCR5 and the consensus operator. Yager’s 
rule was found to produce the lowest combination evidence 
strengths and the largest combination evidence strength ranges. 
It was also observed that the BMA formulation methods which 
used source discounting produced slightly higher combination 
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Fig. 3. Evidence strengths of combined subjective confidences, discounted by evidence strength in the line length task. Results shown for the hardest difficulty. 
Duplicates of the subject which exhibited the lowest evidence strength included in the combination, (a) Average lower bound of the evidence strength range 
versus the number of worst source duplicates present in the combination. Higher evidence strengths are better, (b) Size of evidence strength range versus the 
number of worst source duplicates present in the combination. Smaller evidence strength range sizes are better. 



evidence strengths than the ones which did not. Discounting by 
decision reliabilities and source evidence strengths produced 
similar combination evidence strengths. 

Figure 3 shows the average strength of each combination 
method (namely, Bayes’ Rule, DRC, Yager’s Rule, PCR5 and 
the Consensus operator) when discounting the sources using 
their estimated evidence strengths, and while also including an 
increasingly large number of duplicates of the subject which 
exhibited the lowest evidence strength in the combination (i.e., 
the “worst” source). These subjects were combined in a similar 
manner as in Figure 2, except that the worst source and its 
duplicates were combined after the other five sources. When 
more than 10 duplicates of the worst source were included in 
the combination, Bayes’ rule and the DRC produced higher 
combination evidence strengths than the other combination 
methods (Figure 3a). The size of the evidence strength range 
was found to decrease the fastest with PCR5 and the slowest 
with the consensus operator (Figure 3b). The size of the 
evidence strength range for Yager’s rule was found to stay 
relatively constant. 

VI. Conclusions 

We have demonstrated how it is possible to use a model 
of human decision making and confidence assessment to 
construct a simulator for evaluating the performance of data 
fusion systems using subjective opinions. In our simulations, 
Bayes’ Rule, the DRC, the PCR5, and the consensus operator 
performed similarly when source discounting was performed. 
Both Bayes’ rule and the DRC were unable to combine 
subjective confidences consistently when source discounting 
was not performed. Finally, when including more than 10 
duplicates of an inferior-quality source in the combination, 
Bayes’ rule and the DRC produced higher evidence strengths 
than the other combination methods. Future work includes 
extending the simulator for use on M - ary ( M > 2) f 2, and 
eventually the development of a test bed for hard and soft 
data fusion algorithms for multiple tasks and models of human 
decision makers. 
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